perf: purge in chunks of 8, not 100 - #5134
Closed
mennatnaga wants to merge 1 commit into
Closed
Conversation
mennatnaga
force-pushed
the
menna/purge-chunk-size
branch
from
August 11, 2026 22:45
c7d22ac to
2d208e4
Compare
mennatnaga
changed the base branch from
shinzo-develop
to
menna/p2p-close-race
August 11, 2026 22:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #5133.
purgeChunkSizebounds how many documents share one purge transaction. It keeps a commit under the store's transaction size limit, which it does at 100, but it also sets the cost of the purge and that part was never measured.Each purged document opens several iterators inside the chunk's transaction, and badger rebuilds and re-sorts the transaction's whole pending-write set on every one, so a chunk costs roughly the square of its size and total purge time is close to linear in the constant. It is the same cost
mergeChunkSize = 8was introduced to remove from the merge path.BenchmarkPurgeByDocIDsChunkSize, added here, measures 2,739 docs/s at 8 against 713 at 100 and 365 at 200, on disk with three secondary indexes and history pruning on. A host runs 44 to 203 docs/s on the same code, so the magnitudes do not reconcile and this is a candidate to A/B rather than a proven fix. Sitting at the top of the stack means the treated build differs from what the test mesh already runs by this constant alone. If drain on a treated host stays under 100 docs/s the cost is in the per-document DAG walk, not the chunking.